Skip to content

test(mcp): add HTTP Code Mode agent evaluations - #1675

Draft
betegon wants to merge 1 commit into
bt/code-mode-httpfrom
bt/code-mode-http-evals
Draft

betegon wants to merge 1 commit into
bt/code-mode-httpfrom
bt/code-mode-http-evals

Conversation

@betegon

@betegon betegon commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Prerequisite #1676 is merged. Stack: #1670 (HTTP Code Mode) → this PR. Merge #1670 before rebasing/retargeting this PR to main.

Tests whether agents discover HTTP Code Mode themselves and complete API tasks missing from the normal MCP catalog. Real Claude/Codex clients use local Dynamic Workers and stateful API fixtures; independent verifiers check the resulting state, including collateral writes, rather than trusting the agent's answer.

The harness covers dashboard copying, releases/deploys, repository pagination, alerts, monitors, partial-write recovery and permission failures. A catalog-only control measures the existing capability gap. Reports retain model usage, tool/API calls and provenance. Runtime, permissions and their safety tests stay in #1670; SDK comparisons remain future work.

Restacked onto c50537e7 from #1670. The evaluation patch is unchanged (git range-diff / patch identity verified). 56 fixture/adapter tests + 4 discovery integration tests, both Code Mode typechecks, workspace typecheck/lint, documentation links and frozen install pass on c9a02025.

The full workspace test run on the parent has four failing files in unchanged CLI code; publication with those failures documented was explicitly approved, and no CLI changes are included. No new model runs were performed for this restack; earlier results and their limits remain in the evaluation guide. Fixtures do not establish production authorization or hosted CPU enforcement.

@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-local Ready Ready Preview Oct 30, 2026 5:56pm UTC

Request Review

@betegon
betegon force-pushed the bt/code-mode-http-evals branch from 8d7e92d to 954c0cd Compare October 27, 2026 13:28
betegon added a commit that referenced this pull request Oct 27, 2026
The Cloudflare test plugin still used an older Miniflare/workerd than
Wrangler, so updating the direct dependency alone left different
runtimes in local and integration tests.

Switch to `@cloudflare/vitest-plugin` 1.4.6 and align Miniflare/workerd
with Wrangler. Miniflare 5 removed `fetchMock`; an Undici outbound
service preserves the existing fixtures and `disableNetConnect()`
policy.

This is the prerequisite for #1670 (HTTP Code Mode), followed by #1675
(agent evals). Only seven test/dependency files move here; the broker,
permissions and production integration remain in #1670.

Validated independently on Node 24: **443 Cloudflare tests across 34
files**, root typechecks, lint and frozen install. The earlier full
workspace run had unrelated CLI fixture/platform failures; CLI fixes are
excluded. No deployment performed.
@betegon
betegon force-pushed the bt/code-mode-http-evals branch from 954c0cd to d518be2 Compare October 27, 2026 13:44
@betegon
betegon force-pushed the bt/code-mode-http-evals branch from d518be2 to 6399954 Compare October 27, 2026 17:11

This branch was successfully deployed

2 active deployments
Preview — c9a02025 Deployed Oct 30, 2026 by vercel[bot]
Actions — c9a02025 Deployed Oct 30, 2026 by betegon via eval #1426
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant